Estimating Expected Calibration Errors

Uncertainty in probabilistic classifiers predictions is a key concern when models are used to support human decision making, broader pipelines or sensitive automatic decisions have be taken. Studies shown that most not intrinsically well calibrated, meaning their scores consistent with posterior probabilities. Hence being able calibrate these models, enforce calibration while learning them, has regained interest recent literature. In this context, properly assessing paramount quantify new contributions tackling calibration. However, there room for improvement commonly metrics and evaluation of could benefit from deeper analyses. Thus paper focuses on the empirical context classification. More specifically it evaluates different estimators Expected Calibration Error (ECE), amongst which legacy some novel ones, proposed paper. We build an procedure quality ECE estimators, use decide estimator should practice settings.